According to God of Prompt on Twitter, lowering the model temperature parameter from 0.7 to 0.3 when evaluating with TruthfulQA significantly increases the 'truthful' answer score by 17%, not by improving actual accuracy, but by making models respond more cautiously and hedge with phrases like 'I don't know' (source: twitter.com/godofprompt/status/2011366460321657230). This exposes a key limitation in the TruthfulQA benchmark, as it primarily measures the conservativeness of AI responses rather than genuine accuracy, impacting how AI performance and business trustworthiness are assessed in real-world applications.

According to God of Prompt on Twitter, current AI research practices often emphasize achieving state-of-the-art (SOTA) results on benchmarks like TruthfulQA, sometimes at the expense of scientific rigor and real safety advancements. The tweet describes a case where a researcher ran 47 configurations, published only the 4 that marginally improved TruthfulQA by 2%, and ignored the rest, highlighting a statistical fishing approach (source: @godofprompt, Jan 14, 2026). This trend incentivizes researchers to optimize for publication acceptance rather than genuine progress in AI safety, potentially skewing the direction of AI innovation and undermining reliable safety improvements. For AI businesses, this suggests a market opportunity for solutions that prioritize transparent evaluation and robust safety metrics beyond benchmark-driven incentives.